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the effect of various methodologies, tools, and models on 
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ful development practices. The activities, findings, and 
recommendations of the SEL are recorded in the Software En- 
gineering Laboratory Series, a continuing series of reports 
that include this document. 
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ABSTRACT 


The use of the Ada® language and design methodologies 
that utilize its features has a strong impact on all phases 
of the software development project lifecycle. At the Na- 
tional Aeronautics and Space Administration/Goddard Space 
Flight Center (NASA/GSFC) , the Software Engineering Labora- 
tory (SEL) conducted an experiment in parallel development 
of two flight dynamics systems in FORTRAN and Ada. The teams 
found some qualitative differences between the system test 
phases of the two projects. Although planning for system 
testing and conducting of tests were not generally affected 
by the use of Ada, the solving of problems found in system 
testing was generally facilitated by Ada constructs and de- 
sign methodology. Most problems found in system testing 
were not due to difficulty with the language or methodology 
but to lack of experience with the application. 


lAda® is a registered trademark of the U. S. Government Ada 
Joint Program Office (AJPO) . 
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SECTION 1 - INTRODUCTION 


Ada® 1 is not just a new programming language but a part 
of a major advance in software engineering technology that 
includes new approaches for all phases of the software de- 
velopment lifecycle. This paper, one of a series of reports 
examining each project phase [Brophy 1987, Brophy 1988], 
evaluates the impact of the use of Ada when compared with 
FORTRAN in the system test phases of two projects. 

The Software Engineering Laboratory (SEL) of the National 

Aeronautics and Space Administration/Goddard Space Flight 

Center (NASA/GSFC) conducted an experiment involving the 

parallel development of a software system in both the Ada 

2 

and FORTRAN programming languages. NASA/GSFC and Com- 
puter Sciences Corporation (CSC) were cosponsors of the 
experiment, which was supported by personnel from the three 
SEL participating organizations: NASA/GSFC, CSC, and the 

University of Maryland. The chief goals of the study were 
to characterize the development lifecycle of a large project 
when Ada is used as the implementation language with a de- 
sign methodology that can take advantage of its features and 
to determine the impact of the use of Ada on reusability, 
reliability, maintainability, productivity, and portability. 

Two teams each developed a Gamma Ray Observatory (GRO) sat- 
ellite dynamics simulator from the same specifications. One 
team used FORTRAN as the target language with a conventional 


1 Ada® is a registered trademark of the U. S. Government 
Ada Joint Program Office (AJPO) . 

^The acronyms were Gamma Ray Observatory (GRO) Dynamics 
Simulator in Ada (GRODY) for the Ada project and GRO Dynam- 
ics Simulator in FORTRAN (GROSS) for the FORTRAN project. 


5202 


1-1 



design methodology, which is the usual approach for this 
type of application. The other team used Ada, with an 
object-oriented design methodology developed at NASA/GSFC 
[Seidewitz, Stark 1986] . NASA uses the GRO dynamics simula- 
tor to test and to evaluate GRO flight software under condi- 
tions that simulate the expected in-flight environment as 
closely as possible [Agresti 1986] . By the end of the 
system testing phases, the teams had produced 39,767 lines 
of FORTRAN and 128,046 lines of Ada, where lines of code are 
the total number of physical lines including exe- cutable 
code and nonexecutable code, comments, and blank lines. 
Although these figures give a rough idea of the com- 
parative sizes of the two efforts, they do not give a pre- 
cise basis for comparison of the effort required for 
development in the two languages [Firesmith 1988] . 

Data were collected directly from team members and from a 
database maintained by SEL. Members of both teams who par- 
ticipated in system testing were interviewed and asked about 
their expectations, actual findings, problems, solutions, 
and opinions. Team members also completed forms throughout 
the projects describing their effort levels and changes to 
code, and that information was entered into the SEL database. 
Presented data are taken from the database, and other sources 
are referred to since much of the data has already been 
reported . 
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SECTION 2 - DEFINITION OF THE SYSTEM TEST PHASE 


Ada unit testers performed some integration before system 
testing officially began. System testing and unit testing 
effort overlapped considerably. The team members reported 
their hours on Personnel Resource Forms (PRFs) and attributed 
hours to specific activities. Figures 2-1 and 2-2 show the 
weekly efforts for unit testing and system testing on the 
two projects. 

In the FORTRAN project, a clear delineation exists between 
effort attributed to system testing and effort attributed to 
unit testing although they overlap slightly. In the Ada 
project, participants were performing system test work at 
the same time as unit test work, and the overlap is consid- 
erable. This overlap plus team members’ comments suggest 
that the line between unit testing and system testing was 
blurred on the Ada project. 

When the data from PRFs giving time attributed specifically 
to system testing is considered and this effort is calcu- 
lated as a percentage of total project effort, the Ada proj- 
ect used 11.3 percent of its effort on system testing, and 
the FORTRAN project used 8.91 percent. In addition to de- 
fining activities by the hours attributed directly to them, 
each project phase had a formal start and end date. Regard- 
less of attributed activity, the sum of all effort occurring 
during the system test phases was found and the effort during 
system test determined as a percentage of all effort on the 
entire project. Of the total effort on the two projects, 
the portion used during the system test phase was 22.8 per- 
cent on the Ada project and 17.9 percent on the FORTRAN 
project. The standard allotment for system testing is 
20 percent in the flight dynamics area. The Ada project 
system testing phase was not grossly out of proportion, but 
a general conclusion cannot be drawn about the language since 
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other variables, such as greater training time for the Ada 
project and overlap of activities other than system testing 
in the system testing phases, exist. 
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SECTION 3 - WRITING THE SYSTEM TEST PLAN 


The author of the system test plan for the Ada experiment 
[Stark 1987] said that the plan was based on the FORTRAN 
plan being developed in parallel [Garrick] . He found no 
need for special consideration because of the use of Ada or 
the object-oriented design methodology; this is consistent 
with the idea that system test plans in this environment are 
generally written to test against functional specifications, 
which ideally do not depend on the implementation language. 
However, because the Ada team did not have the same schedule 
constraints as the FORTRAN team, they defined more tests — 31 
compared to 14 for the FORTRAN team. 
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SECTION 4 - CONDUCTING THE SYSTEM TEST 


4.1 IMPACT OF ADA FEATURES 

Conducting system tests was not generally different for the 
Ada project than for the FORTRAN project. The system test 
teams usually did not need to examine internals to run tests 
and to evaluate results. However, the Ada team did find a 
few Ada features that needed special attention. 

One case in which an Ada feature was an issue was in induc- 
ing conditions that would cause Ada exceptions to be raised. 
Many times this inducement was relatively easy, such as de- 
letion of a required file; other times it was not, i.e., for 
exceptions that flagged conditions that may not be intro- 
duced externally such as division by zero. Although some 
exceptions were difficult to test overall, the team felt 
that they aided in comprehensive error handling. 

The Ada test team reported that it was difficult to coordi- 
nate concurrent tasks for testing although this coordination 
can be challenging regardless of the language. The Ada lan- 
guage offers tasking but FORTRAN does not, so the Ada team 
took advantage of the ease of tasking more than the FORTRAN 
team [Brophy 1988] . Although concurrency, was easier to 
design and implement in Ada, the team reported that set- 
ting up tests and diagnosing problems were more difficult. 
They agreed, however, that these problems were not peculiar 
to Ada but would be found in any system using concurrency 
and that since tasking was easier to implement, Ada provided 
a net advantage when using concurrency. 

The FORTRAN project used a form of tasking that was supported 
by the operating system; the method did not provide true con- 
currency but a series of tasks whose execution was controlled 
by logic within the application software. Only one task was 
active at any given time. The FORTRAN team did not report 
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any unusual problems in testing a system with this architec- 
ture and attributed only one or two errors to difficulties 
stemming from their tasking approach. 

Occasionally, the Ada rename feature caused confusion during 
debugging sessions. This was attributed to the debugger's 
failure to incorporate the rename feature rather than to a 
difficulty in the language. When the debugger did not rec- 
ognize the name used to rename a variable, programmers would 
query the debugger for the value of a variable, and if it 
were a name used to rename another variable, they could not 
get the value. This problem was discussed with a member of 
the Digital Equipment Corporation (DEC) Ada team; she said 
she was unaware of the problem and would treat it as a bug. 
She believed the problem should be fixed and that it might 
even be resolved in the next release of the debugger. 

Although the Ada exception handling, tasking, and rename 
features required special attention and caused some prob- 
lems, none was a major roadblock, and the team felt the 
power added by these features outweighed the difficulties. 

4.2 T OOLS 

Ada development is still relatively new, so despite many 
excellent offerings of Ada tools, their availability is 
neither as great as nor as widely known as the tools for the 
more mature FORTRAN environment. The Ada team developed 
software on a DEC VAX/VMS system,^ and DEC offers tools 
that are compatible with Ada for use on the VAX [Schultz 
1988] . The DEC symbolic debugger and Code Management System 
(CMS) were the tools used in system testing. When asked 
what other tools would have been useful, one team member 


1 DEC, VAX, and VMS are registered trademarks of Digital 
Equipment Corporation. 
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suggested that the DEC Performance and Coverage Analyzer 
(PCA) would also have been helpful; other team members re- 
sponded that they did not suggest that other tools were nec- 
essary because they had no information about other available 
tools. Although no clear need was identified for additional 
tools, more information regarding the availability of other 
Ada-oriented testing tools would have been helpful. 

The FORTRAN team also developed their system on a VAX and 
used only a debugger. They felt that tool was sufficient 
for their testing. 
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SECTION 5 - ERRORS DISCOVERED DURING SYSTEM TESTING 


5.1 SOURCE OF DATA 

All team members recorded information for each software 
change on a Change Report Form (CRF) . The CRF describes the 
type of change. Data were examined for changes with a type 
of error correction. When the type of change is an error 
correction, the form also describes the class of error, the 
source of the error, the time to isolate the error, and the 
time to implement the change. This data was entered into 
the SEL database. 

5.2 CLASSES OF ERRORS DISCOVERED DURING SYSTEM TESTING 

Brophy noted that Ada developers in the experiment found 
unit testing to be more difficult for Ada [Brophy 1988] . 
Since the team found isolation of Ada units to be difficult, 
unit testing usually involved combinations of units rather 
than single units. The team members reported that the types 
of errors discovered through this method of unit testing 
were often mismatched data interfaces and con- flicting 
assumptions between internal components, which are errors 
more typical of those discovered in later testing phases of 
conventional FORTRAN projects. Although this in- tegration 
increased unit testing effort, the team believed that it 
made system testing easier. The team members also found 
that the semantic checking performed by the Ada com- piler 
uncovered mismatched calling sequences at compile time that 
would not have been found in FORTRAN until run time. 

The errors described on the CRFs were divided into the fol- 
lowing classes: computational, data value (usage of vari- 

ables), data initialization, external interface, internal 
interface, and logic. Figure 5-1 shows the distribution of 
errors for each project by class of error. 
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Of the total errors found during system testing, internal 
interface errors accounted for 21 percent in the Ada project 
and 29 percent in the FORTRAN project. However, this appar- 
ent difference is not statistically significant. 

Because the Ada system was not intended to become opera- 
tional, managers placed a lower priority on it when assign- 
ing effort to it, and it was difficult to get support that 
the team thought they needed from analysts who had strong 
backgrounds in the specific application. The team attributed 
most errors to misinterpretation of the specifications, such 
as errors in mathematical computation, rather than design 
errors or coding errors. 

The design for the FORTRAN project was largely based on 
stable designs of similar systems already developed, and 
approximately 36 percent of the code was reused from other 
systems. No precedent existed for an Ada system of the type 
being developed; therefore, the design was new, and only 2 
percent of the code was reused from previous systems 
[McGarry, Agresti 1988] . This difference in reuse is another 
variable that may have affected the error profile of the 
FORTRAN project. 

5.3 SOLVING ERRORS FOUND DURING SYSTEM TESTING 
5.3.1 ISOLATING ERRORS 

The Ada system test team reported that in some ways the Ada 
code was easier to debug than similar FORTRAN systems be- 
cause the design methodology controls access to related data 
as opposed to the FORTRAN implementation that exploited large 
COMMON blocks with little control over data access. For the 
same reason the scope of effect of software errors was more 
limited in the Ada implementation. The team reported that 
they generally found errors easily in the Ada implementation 
because of the program structures that are enforced by the 
language. However, the times to isolate causes of errors 
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indicate that the Ada team actually spent more time solving 
errors than the FORTRAN team. The CRFs described time to 
isolate an error defined as the time it took for the respon- 
sible developer to isolate an error and does not include the 
time to determine who is the responsible developer. As shown 
in Figure 5-2, both teams solved most of their errors in less 
than 1 hour; however, the FORTRAN team solved 82 percent of 
errors in less than 1 hour, and the Ada team solved only 58 
percent in less than 1 hour. When the first two categories 
are combined, they show that the proportion of errors solved 
in less than 1 day were similar for both projects; 94 per- 
cent for the FORTRAN project and 96 percent for the Ada proj- 
ect . 

The Ada compiler does semantic checking that spots some er- 
rors that would not be found until testing in a FORTRAN sys- 
tem, so the proportion of easier to solve errors may have 
been reduced in the Ada system test phase. 

The development team found the readability of Ada as com- 
pared to FORTRAN, in part due to more rigid coding standards, 
to be a clear advantage in debugging, except where long var- 
iable names appeared in complex mathematical expressions. 

In some instances, this problem was easily solved by the 
judicious selection of variable names and by renaming varia- 
bles with long names when they were used in such expressions. 

5.3.2 REPAIRING ERRORS 

As shown in Figure 5-3, once the problems were isolated the 
FORTRAN team needed slightly less time to make the changes. 
Although the Ada compiler is more comprehensive and detects * 
some errors earlier, it often requires recompilation of un- 
changed units that are dependent on changed units. Compila- 
tion errors can occur even in unchanged units being 
recompiled. This recompilation was sometimes a significant 
effort, particularly because of the configuration of the Ada 
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system. The Ada implementation decision of nesting versus 
library units had a ripple effect in debugging at the system 
test level; a great deal of recompilation was necessary be- 
fore some coding changes could be tested. This complaint 
also surfaced in the implementation phase [Brophy 1988] . 

5.3.3 NONLANGUAGE DIFFERENCES 

The FORTRAN team members had greater experience in both the 
language in which they were working and in the particular - 
application [McGarry, Agresti 1988] . The Ada team consist- 
ently reported that the single biggest obstacle to effective 
system testing was the lack of availability of people who 
were intimately familiar with the technical aspects of the 
application. Although the Ada team members were experienced 
software developers, having on the average more years of 
software development experience than the FORTRAN team mem- 
bers [McGarry, Agresti 1988] , their lack of experience with 
the specific application made it more difficult for them to 
detect and solve errors than for the FORTRAN team. These 
differences in personnel background may account for the 
ability of the FORTRAN team to isolate and correct errors 
with equal or less effort than the Ada team, despite the 
language advantages described by the Ada team. 
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SECTION 6 - LESSONS LEARNED 


All personnel involved in both projects believed that soft- 
ware development in Ada with an appropriate design methodol- 
ogy is a different experience than the conventional FORTRAN 
development of similar systems. 

Team members have subjectively attributed the many differ- 
ences between the results of the two projects to various 
differences in languages and design methodologies, but too 
many variables exist to be able to clearly assign all effects 
in the system test phases to their causes. Some general 
statements can be made about what was learned. 

Preparing for system testing and executing the tests was__no:t 
affected bv the programming language . The system test plan 
for Ada was essentially the same as the plan for FORTRAN. 

When running the tests, testers were not concerned with the 
language . 

A good repertoire of tools is important . The extra effort 
needed to resolve confusion and software problems due to the 
error in the debugger shows the impact of even minor prob- 
lems with tool software. An organization can most effec- 
tively use its human resources if it has a good tool set and 
actively promotes the use of the tools. 

Ada mav reduce some types of errors . Team members consist- 
ently reported that the Ada compiler detected many of their 
interface errors even before testing began. Objective data 
neither confirms nor contradicts this assertion, but it is a 
reasonable one since the Ada compiler checks for correct 
interfaces, and the FORTRAN compiler does not. 

Ada mav be easier to debug . Team members reported that 
Ada's better readability and the organization of the team's 
design allowed them to find errors more easily than the 
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FORTRAN team. Objective data neither confirms nor contra- 
dicts this assertion, partly because of uncontrolled experi- 
mental variables. 

Recompilation of Ada units can have a significant cost . 
Recompilation issues should be considered short of compro- 
mising the integrity of the design. It is important to con- 
trol the design to avoid unnecessary dependencies that will 
require extensive recompilation in testing phases. 

Definition of test phases for Ada systems is not well de- 
fined . Testing Ada software at the system level is not as 
clearly defined as was presumed at the outset of the project. 
Although the system test plan itself was nearly the same as 
for the FORTRAN project, and it was clear which tests were 
to be designated system tests, it is very difficult to draw 
a hard line between unit testing and system testing. Test- 
ing Ada software must be approached differently than testing 
systems where functions can be easily isolated for testing. 

The differences that could clearly be attributed to the use 
of Ada were generally positive ones, and Ada features with 
negative aspects were either redeemed by their advantages or 
easily mitigated. As the Ada environment matures and as 
developers get more experience, we expect improvements to 
occur in the building and testing of systems built with Ada 
and object-oriented design when compared to methods that are 
still considered conventional. 
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